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Abstract 

Background: Knowledge production and evaluation are two important functions of health research system (HRS). In 
this article, we aimed to reveal the correlation between evaluation of health research organizations and health knowl- 
edge production promotion. 

Methods: A comprehensive evaluation system was developed to evaluate the academic performance of national medi- 
cal science universities on an annual basis. It assess following domains; stewardship, capacity building and knowledge 
production. Measurable indicators for each domain were assigned, a 'research profile' for each department was pro- 
vided. In this study, we compared the results of annually national Health Research System evaluation findings during 
2005-2008. 

Results: The number of scientific articles has been increased from 4672 to 8816 during 2005 to 2008. It is mentionable 
that, the number of articles which has been published in indexed data bases has risen too. This fact could be related to 
directed policy for more international publication of scientific articles from Iran. The proportion of total articles to the 
number of academic members was 1 .14 in 2008, comparing to 0.84 in 2005. It means that this proportion have increased 
about twice (0.7 Vs 0.45) during mentioned time. Moreover, other scientific products such as authored books based on 
domestic researches and cited articles in textbooks have increased according to special attention to knowledge 
production by policy makers. 

Conclusion: We conclude that Health System Research evaluation could be used as a mean for implementing policies 
and promoting knowledge production. 
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Introduction 

Knowledge management is defined by World 
Health Organization (WHO) as, "a set of princi- 
ples, tools and practices that enable people to 
create knowledge, and to share, translate and 
apply what they know to create value and im- 
prove effectiveness" (l).In recent decades, 
knowledge managers have focused on develop- 



ing and enriching knowledge production strate- 
gies (2). In this regard, Health Research System 
(HRS) has a prominent responsibility in scaling 
up knowledge production (3). 
HRS has certain special functions that help fill- 
ing the gap between Research and Development 
(R&D), especially in developing countries (2, 4). 
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One of these main functions is Monitoring and 
Evaluation (5). This function of the HRS is an 
operational component of stewardship and is 
useful as a tool for promoting the other function 
of HRS such as creating and applying research. 
HRS evaluation could be used for expanding 
policies, emerging trends in health research, 
knowledge generation, and technology develop- 
ment and making recommendations for future 
national policies (6). 

Following the stated mission of the World 
Health Organization (WHO) as, "generation of 
appropriate knowledge and its utilization to- 
wards improved health", the Department of Re- 
search and Technology of the MOHME in the 
Islamic Republic of Iran was tasked with the 
responsibility for designing and implementing 
the HRS functions and policies especially moni- 
toring and evaluation (7). The main focus of this 
evaluation system was knowledge production to 
promote the academic performance of the 
researchers with an emphasis on promoting 
international scientific publications. Evaluation 
of research activities in terms of knowledge pro- 
duction was performed based on stakeholders 
agreed criteria. 

The present paper aims to assess the contribu- 
tion of the HRS evaluation on promotion of 
HRS performance and health knowledge 
production. 

Materials and Methods 

During last decade, the Department of Research 
and Technology of the Ministry of Health and 
Medical Education (MOHME) of I.R. Iran (as 
the main policy making organization in 
biomedical and health related research) concen- 
trated on knowledge management promotion. 
Scientific activity is traditionally measured by 
the production of scientific publications, i.e. the 
number of articles published in scientific jour- 
nals. Besides one decisive criterion in the 
realization of research policy is the number or 



volume of valid scientific research studies con- 
ducted by specialists in different fields (4). 
A comprehensive evaluation system was devel- 
oped to evaluate the academic performance of 
national medical science universities on an annual 
basis. Through a collaborative effort the key HRS 
authorities developed certain measures. A pilot 
study was conducted to establish the applicability of 
the evaluation measures adopted. In each year, the 
evaluation procedures were reviewed and renewed. 
Based on certain policies some measures were 
changed in line with stakeholders' views and the 
objective results obtained. "Stewardship", 
"Capacity Building", and "Knowledge Production" 
were three domains that considered in evaluation 
process. (Table 1) shows the Medical Sciences 
University Evaluation Form. 
According to indexing type, article type and the 
place of publication, all documents were classi- 
fied and cleaned in order to avoid double count- 
ing. The articles were divided into 5 types 
including of original research article, review 
article, brief report or short communication, case 
report and letter to the editor. 
The evaluation was carried out by a team 
consisting of research experts at the office of 
undersecretary for research. All team members 
applied the developed measures to Health Re- 
search System in 40 universities of medical sci- 
ences annually. The measures were assessed 
during a meeting with the representative of each 
university. The evaluation forms were filled out 
based on controlling presented official docu- 
ments. 

Results 

Figure 1 shows the results of the 2005-2008 an- 
nual HRS evaluation report. The data presents 
the total number of published articles and medi- 
cal researchers during mentioned time period. 
The total number of published articles has in- 
creased from 4672 to 8816 during mentioned 
period. It is noticeable that during the same pe- 
riod total number of medical researchers has in- 
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creased 1.2%. The results were analyzed accord- 
ing to cited articles in widely recognized data 
bases. Figure 2 provides detailed information 
about the growth in the total number of cited 
articles. 

The number of indexed articles in reliable and 
widely recognized data bases has risen by 127%. 
Although articles sited in ISI/PubMed have de- 
creased in 2008, the number of published arti- 
cles in Iranian journals indexed in ISI/PubMed 
has increased by 36% (from 654 papers in 2007 
to 893 in 2008). Noticeably, the proportion of 
ISI Web of Science/PubMed indexed articles per 
medical researcher has increased from 0.09 to 
0.33. 

Retrieved data from 2005 to 2008 show that the 
trend of published articles cited in other recog- 
nized data bases including Scopus, EMBASE, 
Chemical Abstract, Index Medicus, EMRO and 
Index Copernicus was ascending. It is noticeable 
that the number of articles published in Iranian 
journals indexed in mentioned databases had 
increased by 14%. Also, the above Figure shows 
that the total number of non-indexed articles had 
a descending trend. A closer look at the data re- 
veals that almost, 85% of articles were original, 
7% case reports, 3% short communication pa- 
pers, 3% letter to the editor and 2% review arti- 
cles in 2008. 

In 2008, national medical universities were 
ranked by the scores they received on different 
knowledge production axes. Table 2 shows the 
top ten ranking Iranian medical sciences 



universities in terms of their scores on knowl- 
edge production. 

The comparisons show that in terms of knowl- 
edge production, Tehran University of Medical 
Science stands in the first place. This university 
had 2346 scientific articles in 2008 that covered 
26.6% of total national biomedical articles. Sit- 
ting in the second place is Shahid Beheshti 
University of Medical Science having published 
1426 articles, which comprises 16% of total arti- 
cles related to this university and Isfahan 
University of Medical Sciences University as 
the third rank had 806 scientific articles in 2008 
that have increase about 110% growth in 
publication. It is mentionable that these top ten 
universities covered 72% of our country 
biomedical articles. 

Findings of this study show that the proportion 
of total articles per research project was 0.84 in 
2005 comparing with 1.14 in 2008. Moreover 
the proportion of total articles per medical re- 
searcher has increased about twice (0.7 Vs 0.45). 
In terms of knowledge production, the data col- 
lected in 2008 shows that 14,357 articles were 
presented in scientific colloquiums (9,938 in 
national congresses and 4,419 in international 
congresses) compared to the 6585 in 2005. 
Furthermore, other scientific products such as 
published books, published scientific articles 
witch cited by credible textbooks have increased 
according to special attention to knowledge pro- 
duction by policy makers. 
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Fig. 2: Articles published during 2006-2008 period 



Table 1 : Medical Sciences University Evaluation Form 



Domain 



Subject 



Measure 



Score 



Stewardship 



Capacity 
Building 



Strategic Planning 



Priority Setting 

Ethics committee Activities 

International Congress 
National congress 

Research related Workshops 



Existence of strategic 
program, report of an- 
nual program evalua- 
tion 

Existence of research 
priorities, Percentage 
of research projects in 
line with research pri- 
orities 

Percentage of referred 
proposals to ethic 

committee 
Held international 
congress confirmed 
bycabinet 
Held national congress 
confirmed by Health 

ministty 
Held required work- 
shop based on Re- 
search Council confir- 
mation 



0-100 



0-100 



0-100 



Each case 100 



0-100 



0-150 
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Table 1 : Continued . . . 



Scientific Awards 



Get award from Razi 
or Kharazmi festivals 



First 100 
Second 70 
Third 50 
Student award 50 



Book authoring 



Citation to articles in textbooks 



Innovations and patents 



Paper presentation in national and 
international congresses 



Knowledge 
Production 



Indexed articles in ISI/Web of Science 
and PubMed/Medline 



Indexed articles in other international 
indexing databases(such as Scopus 
and Chemical Abstract and etc.) 



Nonindexed published articles in na- 
tional and international scientific jour- 
nals 



Books which at least 
1 % of their references 
are authors self cita- 
tions 

Citation to articles in 
credible textbooks 

Practical innovations 
and patents reported in 
scientific journals 



Paper presentation in 
oral or poster form, in 
national and interna- 
tional congresses and 
papers indexed in ISI 
proceedings 



Indexed articles in 
PubMed/Medline and 
ISI/Web of Science by 
considering the impact 
factor with two coeffi- 
cients as the quality 
criteria of a journal. 

Indexed articles in 
other international in- 
dexing databases ; 
Scopus and Chemical 
Abstract, Biological 

Ab- 
stract ,embase,Index 
copernicous,IMEMRO 

Non-indexed pub- 
lished articles in jour- 
nals witch have 
accredited by the Ira- 
nian Ministry of Sci- 
ence, Research and 
Technology (MSRT) 
and the Ministry of 
Health and Medical 
Education (MOHME) 
and credible foreign 
journals 



Each case 25 



Each case 15-25 

National cases: 2-10 
International cases: 5- 
100 

Presented paper in 
national congress: 

Each case 0.5 
Presented paper in 
Internationalcongress: 
-Oral presentation 3 
-Poster presentation 
0.5 

-ISI Abstract Meet- 

ing:2-3 
-ISI Proceeding:3-4 
-Invited Speaker 6 

Each article indexed 
in ISI/WOS :25 
+2(Impact Factor) 
Each article indexed 
in PubMed/Medline: 
25 



Each case 10-15 



Each case 5 
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Table 2: Knowledge production scores of top Ten Iranian Universities based on the 2008 evaluation results 





Medical 

Science 

University 




2005 






2008 




NO. 


Academic 
Member 


Total 
Articles 


Knowledge 
Production 
Score 


Academic 
Member 


Total Ar- 
ticles 


Knowledge 
Production 
Score 


1 


Tehran 


1236 


1156 


18178 


1265 


2346 


43682 


2 


Shahid 
Beheshty 


1049 


656 


9474 


1115 


1426 


25711 


3 


Isfahan 


637 


382 


5976 


611 


806 


13778 


4 


Shiraz 


538 


299 


5373 


656 


746 


13494 


5 


Tabriz 


553 


269 


3797 


607 


647 


11269 


6 


Mashhad 


584 


320 


4085 


629 


562 


9422 


7 


Iran 


721 


245 


3353 


696 


529 


8043 


8 


Baghiatal- 
lah 


183 


185 


2583 


199 


368 


5979 


9 


Ahvaz 


420 


129 


1678 


472 


317 


5209 


10 


Mazanda- 
ran 


209 


187 


2529 


201 


222 


4063 



Discussion 



The National Iranian Plan for Health Innovation 
and Science Development aims to increase the 
capacity to create and facilitate knowledge shar- 
ing and dissemination as two main policies (8). 
The Department of Research and Technology in 
the Ministry of Health and Medical Education 
has been developing systematic plans to realize 
these policy objectives in medical sciences 
universities. Furthermore, supporting national 
programs for knowledge management, promot- 
ing knowledge generation and establishing 
knowledge pattern are national level strategies 
recommended by WHO (2). 
Health research has been broadly defined as "the 
generation of new knowledge using the scien- 
tific method to identify and deal with health 
problems" (9). Generated Knowledge will en- 
able research stakeholders to improve 
implementation of Health research in order to 
promote health outcomes and health equity (3, 
10). 



Significant increase in number of scientific pub- 
lished papers has improved Iran's scientific 
ranking word wide (11, 12). According to ESI 
(Essential Science Indicators), among a five- 
year period (2005-2009) Iran stands in the 19th 
rank for scientific output (13). This promotion 
could be as a result of the systematic approach 
to research management which enables research 
stakeholders to improve health outcomes and 
health equity (3, 10). Increased attention of 
health research policy makers to research pro- 
jects in terms of quality as well as quantity, 
could contribute to raising the ratio of articles 
per projects (14). 

According to the findings of this study the num- 
ber of cited articles in ISI/PubMed has increased 
1.3 times from 2005 to 2008. This fact could be 
as a result of revising research polices which 
offered more credit to the quality rather than 
sheer quantity. The number of Iranian biomedi- 
cal journals has increased from 8 to 155 (1979 to 
2010) (15). The number of Iranian medical jour- 
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nals that are indexed in scientific database in- 
creased from 53 in 2000 to 141 in 2008 (16). 
Increased accessibility to scientific data bases, 
advances in information and communication 
technology (ICT) and indexing Iranian journals 
in credible databases have promoted Iranian 
publication output (17). 

The proportion of articles to academic member 
among, Tehran, Shahid Beheshti and Isfahan 
medical sciences universities were respectively 
1.8 and 1.2 and 1 .3 .Ascending trend of this 
proportion among top universities leads to 
achievement of the comprehensive Medical Plan 
objectives. 

In conclusion we believe that planned policies 
could be safely applied through HRS evaluation 
criteria. Therefore national policies may flow 
from the past, present and future by efficient use 
of available resources. 
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